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Abstract 

A  decision-maker  (e.g„  the  Nuclear  Regulatory  Commission)  seeks  an  expert’s  probabilities  for 
uncertain  quantities  of  interest  (e.g.,  a  seismologist’s  forecast  of  earthquakes),  and  wants  the 
expert’s  reward  to  depend  on  the  accuracy  of  the  predictions.  Assume  that  the  expert  compares 
compensation  schemes  on  the  basis  of  the  expected  utility  of  the  dollar  payoffs,  and  is  willing  to 
reveal  his  utility  function  for  money.  A  reward  is  called  'proper*  if  the  expert  is  never 
encouraged  to  state  probabilities  he  does  not  truly  believe.  It  is  "strictly  proper"  if  he  is,  in  fact, 
encouraged  to  state  his  beliefs. 


The  reward  procedure  suggested  in  this  paper  uses  the  expert’s  stated  probabilities  and  utility 
function  to  select  from  a  set  of  possible  payoffs.  This  procedure  is  always  proper,  but  may  not 
be  strictly  proper.  If  the  preferred  payoff  is  independent  of  the  outcome  whenever  the 
decision-maker  and  expert  agree  on  the  probabilities,  then  they  are  said  to  be  'jointly  risk- 
averse/  (For  example,  if  the  decision-maker  agrees  to  play  'bookie'  to  a  risk-averse  expert, 
then  they  are  jointly  risk-averse.)  In  this  case,  the  reward  is  shown  to  be  strictly  proper,  as  long 
as  they  don’t  disagree  too  much,  so  the  expert  can  gain  from  researching  the  problem  and  care¬ 
fully  assessing  his  probabilities.  In  addition,  the  expert  would  prefer  to  make  the  bet  more 
detailed,  distinguishing  between  finer  grain  events,  whenever  such  detail  exposes  new 
differences  of  opinion. 
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An  Incentive  Approach  to  Eliciting  Probabilities 

Ross  D.  Shachter 

Department  of  Industrial  Engineering  and  Operations  Research 
University  of  California,  Berkeley 


1.  Introduction 

A  decision-maker  wants  to  purchase  an  expert’s  probabilities  for  some  uncertain  events.  As  an 
example,  consider  the  case  of  the  Nuclear  Regulatory  Commission  reviewing  the  license  appli¬ 
cation  for  a  nuclear  power  plant  near  a  fault  line  in  California.  A  critical  factor  in  their  analysis 
may  be  the  probability  of  a  major  earthquake  near  the  facility.  They  would  like  to  contract  with 
a  seismologist  to  research  the  likelihood  of  such  a  quake. 

The  need  for  a  procedure  to  reward  the  expert  for  his  probabilities  is  illustrated  by  the  follow¬ 
ing  exchange.  Professor  Bruce  Bolt  at  U.  C.  Berkeley  has  been  a  strong  voice  for  earthquake 
awareness  and  preparation  in  California.  He  warns  that  [1]  ”...  SO-SO  odds  can  be  viewed  as  a 
modest,  perhaps  even  conservative  appraisal  of  the  likelihood  that  California  will  experience  an 
earthquake  of  magnitude  7  or  greater  during  the  next  ten  years.”  Professor  Richard  Barlow  of 
Berkeley  has  replied  [2],  ”In  many  cases,  though  perhaps  not  in  this  instance,  a  probability  of 
one-half  is  used  when  the  forecaster  has  little  knowledge  concerning  the  event  in  question.” 

Prof.  Barlow,  trying  to  analyze  the  risks  to  the  power  plant,  finds  Prof.  Bolt’s  statement  too 
vague  and  fuzzy.  There  is  no  doubt  regarding  Bolt’s  qualifications  to  predict  and  study  seismic 
activity.  It  would  be  useful,  therefore,  if  he  could  offer  a  more  exact  statement  of  the  probabil¬ 
ity. 

In  this  paper,  a  procedure  is  developed  that  would  encourage  an  expert  to  think  about  and 
express  his  precise  beliefs  when  stating  his  probabilities.  From  a  decision-theoretic  viewpoint,  a 
probability  is  defined  in  the  context  of  comparing  alternatives  in  an  uncertain  environment  [3, 
4].  When  the  expert  is  exposed  to  risk  under  uncertainty,  he  has  a  real  need  to  think  in  terms 
of  probabilities. 

This  suggests  a  payment  to  the  expert  for  his  probabilities  that  depends  on  what  he  states  and 
what  actually  occurs.  A  'proper  scoring  rule”  is  a  penalty  function  which  the  expert  can  minim¬ 
ize  in  expectation  by  assessing  the  probabilities  he  truly  believes.  A  scoring  rule  is  "strictly 
proper”  if  that  minimum  is  unique.  A  probability  can  thus  be  defined  with  respect  to  a  scoring 
rule  (5,  6].  Necessary  and  sufficient  conditions  for  a  scoring  rule  to  be  strictly  proper  were 
given  by  Savage  [7]  and  formalized  by  Haim  (8). 

The  expert  is  assumed  to  have  a  utility  function  for  money  and  limited  assets,  and  to  prefer 
that  reward  which  maximizes  his  expected  utility  on  this  venture  only.  This  will  not  coincide 
with  minimizing  his  expected  score  unless  the  utility  function  is  linear  and  the  expert  is  quite 
wealthy,  particularly  when  the  amount  of  money  involved  is  substantial.  Therefore,  if  we 
desire  a  large  enough  incentive  to  encourage  the  expert  to  assess  his  probabilities,  scoring  rules 
will  not  work  in  general.  It  is  necessary  to  incorporate  the  expert’s  utilities  into  the  compensa¬ 
tion  scheme  (9,  10, 11]. 

A  reward  will  be  called  'proper”  if  the  expert  does  not  prefer  to  state  probabilities  he  does  not 
believe.  A  "strictly  proper  reward”  encourages  him  to  say  what  he  believes.  Rather  than  giving 
the  expert  a  score  to  minimize,  the  procedure  developed  in  this  paper  arrives  at  a  reward  func¬ 
tion  following  negotiations  between  the  decision-maker  and  the  expert.  Such  a  reward  will 
always  be  proper,  but  it  may  not  be  strictly  proper. 

Consider  a  decision-maker  and  an  expert  who  are  not  interested  in  betting  on  some  events 


when  they  agree  on  the  probabilities  for  those  events.  Together  they  are  called  "jointly  risk- 
averse.”  This  is  a  generalization  of  the  concept  of  risk-aversion  for  an  individual  [12,  131. 
When  they  don't  agree,  they  prefer  to  bet,  and  their  disagreement  is  an  economic  good,  from 
which  the  expert  expects  to  gain.  If  they  don’t  disagree  too  much,  then  the  negotiated  reward 
is  strictly  proper  [14], 

Suppose  that  the  decision-maker  would  like  to  make  the  bet  more  detailed  by  splitting  one  of 
the  events  into  two  or  more  subevents.  The  expert  prefers  the  new  bet  if  it  exposes  new 
differences  of  opinion  between  them.  This  result  can  be  used  to  obtain  the  expert's  joint  and 
marginal  probability  distributions  for  uncertain  quantities. 

2.  Example  of  Procedure 

In  this  section,  we  consider  an  example  of  a  decision-maker  purchasing  an  expert’s  probability 
for  an  uncertain  event  £,  "a  major  earthquake  will  occur  in  California  within  the  next  ten 
years,"  and  its  complement  £.  Suppose  that  the  decision-maker  expects  to  pay  20,000  dollars 
ten  years  from  now  for  the  expert’s  opinion,  but  is  willing  to  pay  up  to  100,000  dollars  for  any 
outcome  if  necessary  to  obtain  the  expert’s  best  assessment.  The  decision-maker  thinks  that 

the  probability  of  an  earthquake  is  q  -  y,  and  will  agree  to  any  payoff  x  -  (xi,x2)  €  X  where 

x\  is  the  payment  to  the  expert  when  an  earthquake  occurs  and  x2  is  the  payment  otherwise: 

2T-  {  x  €  R2  :  j  x,  +  j  x2  <  20000;  x,  <  100000;  x2  <  100000  } 

The  decision-maker  may  be  thought  of  as  playing  the  role  of  a  bookie.  The  expert  has  20,000 

dollars  to  spend  on  bets.  Each  dollar  bet  on  event  E  pays  —  —  3  dollars,  should  £  occur,  each 

-  1  3  * 

dollar  bet  on  event  £  pays  —  -  ■=■  dollars  if  E  does  not  occur. 

l-q  2 

The  expert  in  this  case  is  a  seismologist,  who  we  assume  will  follow  the  axioms  of  decision 
analysis.  If  he  has  utility  function  for  money  u,  assets  a„,  and  believes  that  the  probability  of 
an  earthquake  is  p,  then  he  should  select  that  payoff  in  X  which  maximizes  his  expected  utility, 
given  by 

p  u(  a„  +  x i )  +  (1— p)  u(  a,  +  x2 )  . 

If  we  further  assume  that  the  expert  would  not  like  to  risk  any  of  his  current  holdings,  then 
a,  -  0  and  his  expected  utility  for  the  preferred  payoff  is  given  by 

max  {  p  u<X|)  +  (1  -p)  u(xj)  :  x  €  X,  X|  >  0,  xj  >  0}  . 

The  selling  price  or  certainty  equivalent  of  the  preferred  payoff  is  an  amount  such  that  the 
expert  is  indifferent  between  that  amount  for  certain  and  the  preferred  payoff. 

Consider  a  constantly  risk-averse  expert  [13],  with  utility  function 

i/(x)  -  -  e~*M ,  x  >  0  . 

In  this  case,  if  we  apply  Theorem  2  from  Section  6, 

X|  —  max{  0  ,  mini  60000 , 20000  +  -p-  log  (  ) }  J 

m  l~y 

and 

xj  -  max{  0  ,  mini  30000 , 20000  +  3“ ,0*  <  ) ) )  • 

If  he  selects  an  interior  point,  then  his  expected  utility  is 

1  1  — 

p  w(x|)  +  (1-p)  u(x2)  -  w (20000)  Op) 3  [  |-  (1-p)  ] 3  >  u(20000) 


with  selling  price 


«-‘l  p  «(*,)  +  (1  -p)  u(x2)  )  -  20000  +  -r-  log  [  Op)  (  T  (1-p)  )2 1  >  20000  . 

The  two  inequalities  are  strict  unless  p  -  q  -  y.  Thus,  there  are  two  cases.  Either  the 

decision-maker  and  expert  agree  on  the  probabilities  in  which  case  the  expert  receives  20,000 
dollars  for  certain,  or  they  disagree,  and  the  expert  receives  more  satisfaction  by  betting  against 
the  decision-maker  and  exploiting  their  difference  of  opinion. 

In  Table  I,  the  expert’s  preferred  payoffs  are  shown  as  a  function  of  his  probability  p.  The 
expert  is  assumed  to  have  constant  risk  aversion  a  -  10-4.  Note  that  there  is  a  unique  payoff 
for  every  probability  p,  .0243  <  p  <  .995.  The  decision-maker,  who  believes  that  the  proba¬ 
bility  is  -p  always  expects  to  pay  20,000  dollars.  The  expert  finds  the  payoff  worth  at  least  that 

much,  and  worth  the  most  when  he  disagrees  most  with  the  decision-maker.  Those  are  the 
times  when  his  opinion  may  be  considered  most  valuable  to  the  decision-maker. 

Table  II  shows  the  sensitivity  of  these  results  to  the  risk-aversion  of  the  expert.  When  the 
expert  is  most  risk-averse,  his  betting  is  timid,  and  his  disagreement  must  be  extreme  before  he 
will  bet  all  20,000  dollars.  As  a  result,  there  is  the  widest  range  of  probabilities  for  which  his 
preferred  payoff  is  unique.  On  the  other  hand,  if  the  expert  is  least  risk-averse,  he  may  bet  all 
he  has  even  if  his  disagreement  with  the  decision-maker  is  modest.  Since  the  choice  here  is  so 
sensitive  to  his  probabilities  and  the  stakes  may  be  substantial,  he  should  weigh  his  forecast 
carefully. 

3.  Reward  and  Payoff  Functions 

A  decision-maker  considers  m  mutually  exclusive  events  £|,  •••,£*,  and  their  union, 

m 

F  =  £,,  so  that  (£|,  ••• ,  Em)  is  a  partition  of  F.  The  decision-maker  would  like  the 

#*>i 

expert’s  reward  to  depend  on  both  his  stated  probabilities  and  the  observed  event.  The  reward 
is  then  an  uncertain  venture,  which  entails  the  payment  of  x,  dollars  when  event  E,  occurs  (or 
'  receipt  of  (-*,)  dollars  if  x,  is  negative)  and  the  m-vector  payoff  x  -  (xi,  •  •  • ,  xm)  denotes 
the  payment  to  the  expert  when  event  F  occurs.  When  F  does  not  occur,  the  decision-maker 
pays  a  side  payment  of  c  dollars. 

Definition.  A  reward  R  is  a  function  R  .  P  —>  X  where  P  is  the  set  of  probabilities 
P  -  (  p  €  Rm  :  J^p,  -  1;  p,  ^  0  for  /  -  1,  •••,/») 

and  X  C  Rm  is  a  compact  set  of  possible  payoffs  to  the  expert.  The  expert  states  probabilities 
p  €  P.  If  event  F  occurs,  then  exactly  one  event  E,  occurs,  and  the  decision-maker  pays  the 
expert  /?<(/>)  dollars.  The  real  number  c  is  a  side  payment  if  the  decision-maker  pays  the 
expert  c  dollars  when  the  event  F  does  not  occur. 

Definition.  The  reward  R  is  proper  if  the  expert  never  prefers  to  state  probabilities  different 
from  his  true  beliefs. 

Definition.  The  reward  R  is  strictly  proper  at  probabilities  p  if  the  expert  prefers  to  state  his 
true  beliefs  when  he  believes  p.  A  reward  is  strictly  proper  over  a  set  if  it  is  strictly  proper  at 
all  points  in  the  set. 

Consider  the  constant  reward  R(p)  -  (c,  •  •  • ,  c).  It  is  proper  but  not  strictly  proper. 

Definition.  A  payoff  function  Y  is  a  point-to-set  mapping  Y  \  R  —  X  where  X  <Z  Rm  is  the 
set  of  possible  payoffs.  It  will  be  assumed  that  Y(c)  £  Y(d)  if  c  <  d. 

Definition.  Re  is  an  admissible  reward  if,  should  the  expert  believe  probabilities  p,  there  is 
no  payoff  in  the  set  K(c)  which  he  prefers  to  Re{p). 

Claim.  Admissible  reward  Re  is  proper. 


Table  II.  The  expert’s  preferred  payoff  for  different  risk-aversions.  The  expert  is  con¬ 
stantly  risk-averse  and  the  decision-maker  pays  $20,000  in  expected  value,  and  no  less 
than  zero  dollars  on  any  outcome. 


Risk  Aversion 

.001  .0001 

.00001 

.000001 

Selling  Price  of  a  50:50 
chance  at  $20,000  or  $0. 

693  5662 

9501 

9950 

Range  of  p  for  which  expert 

IE- 13  .0243 

.270 

.3267 

does  not  bet  all  $20,000. 

1-1E-26  .995 

.477 

.3468 

Probabilities  .01,  .99 

Payoff 

E 

not  E 

17399  0 

21301  30000 

0 

30000 

0 

30000 

Selling  Price 

20905  28253 

29651 

29696 

Probabilities  .1,  .9 

Payoff 

E 

18997  9973 

0 

0 

not  E 

20501  25014 

30000 

30000 

Selling  Price 

20201  22013 

26561 

26959 

Probabilities  .3333,  .6667 

PayofT 

E 

20000  20000 

20000 

20000 

not  E 

20000  20000 

20000 

20000 

Selling  Price 

20000  20000 

20000 

20000 

PayofT 
Selling  Price 


Probabilities  .5,  .5 


E 

20462 

24621 

60000 

60000 

not  E 

19769 

17690 

0 

0 

20057 

20566 

25566 

29550 

Probabilities  .99, 

.01 

E 

23526 

55255 

60000 

60000 

not  E 

18237 

2372 

0 

0 

22437 

44369 

59181 

59382 

PayofT 
Selling  Price 
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4.  Assumptions  of  the  Procedure 

The  procedure  is  based  on  the  following  assumptions  about  the  behavior  of  the  decision-maker 
and  the  expert  relative  to  the  mutually  exclusive  events  £|,  •  ,  Em.  It  is  hoped  that  in  some 

contexts  these  assumptions  will  seem  quite  reasonable  and  the  procedure  may  prove  useful. 
There  are  situations  where  these  assumptions  are  not  so  acceptable.  In  those  cases,  there  may 
be  no  good  way  to  reward  an  expert  for  his  probabilities. 

1.  The  decision-maker  can  set  a  point  in  time  at  which  he  will  know  which  event  has  occurred. 

This  requires  that  the  experiment  has  an  unambiguous  and  indisputable  outcome.  The  pro¬ 
cedure  seems  most  effective  when  that  outcome  will  be  known  soon. 

2.  The  expert  is  willing  to  bet  up  to  a„  dollars,  and  the  decision-maker  a¥  dollars  on  any 
specified  outcome.  The  set  of  possible  payoffs  is 

X  —  {  x  €  Rm  :  — <  xt  <  a,  for  i  —  1,  •  •  • ,  m)  . 

Both  the  decision-maker  and  expert  should  be  willing  to  spend  an  amount  independent  of  the 
outcome.  While  it  is  expected  that  one  outcome  may  leave  the  decision-maker  in  a  better  asset 
position  than  another,  the  procedure  is  inherently  a  "zero-sum"  game.  This  assumption  elim¬ 
inates  the  risk-sharing  aspects  possible  with  variable  payoffs,  in  order  to  concentrate  on  the  flow 
of  information. 

The  expert’s  asset  position,  aB,  should  not  depend  on  the  possible  outcomes.  As  an  example, 
consider  our  seismologist  living  near  a  major  fault.  He  may  agree  completely  with  the 
decision-maker’s  probabilities  and  want  to  bet  nonetheless  as  a  form  of  earthquake  insurance. 
It  is  theoretically  possible  to  incorporate  such  conflict  of  interest  in  the  analysis,  but  would  be 
difficult  to  implement.  It  seems  more  reasonable  to  rely  on  the  expert  to  be  accurate  and  objec¬ 
tive  to  establish  and  protect  his  reputation. 

3.  The  expert  believes  probabilities  p  -  (pi,  •  •  • ,  pj  and  utility  function  u  :  R  —  R  such 

that  he  prefers  payoff  x  6  X  to  payoff  y  6  X  if  «(  au  +  x, )  >  «(  au  +  y,  ).  He  is 

Hi  Hi 

m  m 

indifferent  between  them  if  £p,  u(  au  +  x, )  —  £p/  u(  au  +  y, ).  The  marginal  utility  u'  is 

Hi  Hi 

continuous  and  positive  over  (0,au  +  av). 

The  expert's  preferences  between  payoffs  are  characterized  by  his  probabilities  p  and  his  utility 
function  for  money  u.  p,  is  his  subjective  conditional  probability  for  E,  given  F,  and  it  is 
unique.  The  utility  function  is  not  unique  since  any  positive  linear  transformation 
a  u(x)  +  b,  a  >  0,  reveals  the  same  preferences  as  u. 

The  assumption  that  there  exists  such  a  utility  function  of  money  is  strong.  The  expert’s  satis¬ 
faction  at  the  outcome  is  determined  solely  by  the  payment  he  will  receive.  If  x,  -  xJ%  then  the 
expert  should  not  care  whether  it  was  event  £ )  or  Ej  which  occurred.  If  the  expert  has  a  stake 
in  the  different  outcomes  or  the  decision-maker’s  decisions,  then  he  may  not  be  encouraged  to 
reveal  his  true  beliefs.  While  this  assumption  may  be  difficult  for  an  expert  to  satisfy  precisely, 
that  may  not  be  necessary.  He  is  not  in  an  adversary  role  with  the  decision-maker,  and  should 
be  willing  to  cooperate  in  revealing  his  true  probabilities,  provided  he  does  not  suffer  as  a 
result. 

The  requirements  on  the  marginal  utility  are  for  mathematical  convenience.  If  u'  is  positive, 
then  the  expert  is  never  saturated  with  money  and  always  gets  additional  satisfaction  from  a 
larger  payment. 

4.  The  decision-maker  offers  payoff  function 

Y(c)  -  {  €  X  :  (  a¥  -  x, )  >  v(  av  -  c  )  } 

h 

where  —a,  <  c  <  a,.  He  states  i^nzero  probabilities  q  *■  (?i.  •  ■  • ,  qm)  and  utility  function 
¥  :  R  -*  R,  with  marginal  utility  v'  continuous  and  positive  over  (0,a„  +  at). 


The  decision-maker’s  q  and  v  need  not  reflect  his  true  beliefs.  Thty  are  revealed  at  the  start  of 
the  process  before  a  side  payment  c  is  negotiated.  If  the  expert  finds  them  unacceptable,  he 
may  choose  not  to  participate  in  the  procedure.  It  is  doubtful  that  the  decision-maker’s  true 
beliefs  could  in  fact  be  incorporated  in  a  utility  function  independent  of  the  outcome.  If  it  is 
worth  his  white  to  consult  the  expert  in  the  first  place,  then  he  most  likely  stands  to  gain  or 
lose  substantially  from  the  events  in  question.  Regardless,  the  expert,  in  selling  his  knowledge, 
should  be  insulated  from  such  risks. 

Ideally,  q  and  v  should  be  chosen  to  maximize  the  expected  utility  of  the  sample  information. 
In  practice  that  would  be  difficult  to  analyze.  A  reasonable  q  may  be  the  decision-maker’s  true 
beliefs.  If  he  thinks  that  the  expert  is  "biased,”  then  he  may  want  to  bias  his  q  in  the  same 
direction.  If  the  decision-maker  chooses  a  linear  function  v  then  he  is  asking  the  expert  to  bet 
against  a  bookie.  This  is  appealing  in  its  simplicity  and  may  be  realistic  if  the  decision-maker 
has  considerably  more  assets  than  the  expert,  and  is  approximately  risk-neutral  over  the  range 
of  possible  payoffs.  If,  on  the  other  hand,  the  decision-maker  is  conservative,  and  reluctant  to 
pay  much  more  than  c  dollars,  he  may  want  to  state  a  risk-averse  v.  The  effects  of  different 
risk-aversions  on  the  admissible  reward  are  shown  in  Table  II. 

5.  The  decision-maker  and  the  expert  agree  to  side  payment  c,  ~au  <  c  <  a„  to  be  made  at 
such  time  as  it  will  be  known  which  event  has  occurred. 

This  ensures  that  the  decision-maker  is  willing  to  pay  enough  to  obtain  the  expert’s  services. 
Under  these  assumptions,  the  admissible  reward  Rc(p)  is  a  solution  to 

m 

maximize  I A  u(  a„  +  x,) 

/■l 

m 

subject  to  £ q ,  v(  a,  —  x, )  ^  v(  a¥—  c) 

Hi 

-au  <  x,  <  a,  for  i  —  1,  •  •  • ,  m. 

This  is  precisely  the  "Pareto  optimal"  reward  when  two  people  bet  on  uncertain  events  in  which 
they  have  no  stake. 

5.  The  Procedure 

The  procedure  is  now  completely  determined: 

1.  The  decision-maker  states  events  £|,  •  •  • ,  Em  utility  function  v,  assets  a„  and  probabili¬ 
ties  q.  A  point  in  time  is  set  when  it  will  be  known  which  event  has  occurred  so  that  the  payoff 
can  be  made. 

2.  The  expert  states  utility  function  u  and  assets  ar 

3.  They  agree  to  side  payment  c. 

4.  The  expert  states  probabilities  p. 

5.  After  observing  event  Eh  the  decision-maker  pays  the  expert  Rf(p)  dollars.  If  event  F  does 
not  occur,  then  the  decision-maker  pays  him  c  dollars. 

Cdnsider  two  examples  where  the  decision-maker  states  a  linear  utility  function  and  thus  acts  as 
a  bookie.  For  simplicity,  assume  that  they  both  have  enough  assets  so  that  the  admissible 
reward  is  in  the  interior  of  X.  The  admissible  reward  must  satisfy  the  first  order  conditions  for 
the  mathematical  program  given  in  section  4.  The  general  formula  is  given  in  Theorem  2  of 
section  6. 

Suppose  that  the  expert  has  constant  risk  aversion  a,  and  utility  function  u(x)  —  -e~ax .  Then 

*,'</>)- c+±- togl^-fi 

«  4/  “  4/ 


f  p,  u(Rf(p ))  -  uic)  n(— )*'  >  u(c) 
“  M  9l 


His  expected  utility  is 


(since  u(c)  <  0),  and  his  selling  price  for  the  reward  is 

M~'(  Tp,  u(R}'(p))  ]-£•+—  log  (n(— )"'  1  ^  C  . 

|T|  “  -I  P, 

The  two  inequalities  are  strict  unless  p,  -  q,  for  /  -  1,  •  ■  • ,  m.  Either  they  agree  completely 
and  the  payment  is  c  dollars  no  matter  what  happens,  or  the  expert  gains  satisfaction  by  betting 
against  the  decision-maker.  That  gain  comes  from  their  difference  of  opinion. 

Suppose  instead  that  the  expert  has  logarithmic  utility,  uix)  -  log(x  +  $),  x  ^  0  >  —s.  Then 
the  admissible  reward  is 


Rf(p)  —  —  (c  +  s)  —  s 

Qi 


and  his  expected  utility  is 


£ Pi  u(Rf(p))  -  u(c)  +  log  (  ^  u(c)  . 

/-i  /-I  Pi 

That  reward  has  selling  price 

w"'(  u(Rf(p))  1  -  (c  +  s)  fj(—  )Pt  -  s  ^  c  . 

i-t  i-t  Pt 

Again  the  two  inequalities  are  strict  unless  p,  —  q,  for  /'  —  1,  •  •  • ,  m.  They  either  agree  com¬ 
pletely  and  the  reward  is  always  c  dollars,  or  the  expert  gains  by  expressing  his  disagreement. 

In  both  of  these  examples  the  reward  is  constant  when  they  agree  and  otherwise  the  expert 
gains  satisfaction  by  betting.  When  v  is  linear,  then  any  risk-averse  expert  should  behave  this 
way.  In  general,  when  this  is  true,  the  decision-maker  and  the  expert  will  be  called  "jointly 
risk-averse." 


6.  Joint  Risk-Aversion 

Definition.  The  decision-maker  and  the  expert  are  jointly  risk-averse  if  the  admissible  reward 
when  p  -  q  is  unique,  and  Rc(q)  -  (c,  •  •  • ,  c). 

Theorem  1. 

The  following  statements  are  equivalent: 

*  The  decision-maker  and  the  expert  are  jointly  risk-averse. 


u'(  au  +  c  ) 


v’(  av  -  c  ) 

u"(  au  +  c  ) 
u'(  au  +  c  ) 


is  strictly  decreasing  in  c. 


v"(  av  -  c  ) 
v'(  a„  -  c  ) 


>  0  for  almost  all  c. 


(Proofs  for  this  theorem  and  the  others  appear  in  Shachter  {141.) 

Suppose  that  the  decision-maker  and  the  expert  can  agree  on  the  probability  that  a  given  coin 
will  land  heads-up.  They  are  jointly  risk-averse  if  and  only  if  they  always  prefer  a  constant 
payofT  on  the  outcome  of  a  flip.  Otherwise,  they  would  be  interested  in  betting  on  the  flip, 
even  though  they  agree  on  the  probabilities. 

If  the  expert  is  risk-averse  and  the  decision-maker  states  a  linear  or  concave  function  v,  they 
are  jointly  risk-averse.  Even  when  the  expert  is  risk-prone,  they  can  still  be  jointly  risk-averse 
if  the  decision-maker  is  sufficiently  risk-averse. 
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Theorem  2. 

Suppose  that  the  decision-maker  and  the  expert  are  jointly  risk-averse.  Then  the  following 
statements  are  true: 

•  Admissible  reward  Rcip)  is  a  continuous  function,  for  all  c,  —au  <  c  <  a»: 


Rfip) 

where  b  is  defined  by 


and  X  is  chosen  such  that 


max  {  -au  min  {  av  b(  X  —  )  }  ! 

Pi 


b[ 


u’i  au  +  c) 
v'i  a,-c  ) 


c 


vi  a,  —  Rfip)  )  -  v  i  a„  -  c  )  . 

i- 1 


•  Given  x,  —  au  <  x,  <  a,  for  i  —  l,  •••,«,  then  there  is  exactly  one  vector  of  probabili¬ 
ties  p  and  side  payment  c  such  that  Rc(p)  -  x. 

•  If  —  a„  <  Rfip)  <  a,  for  some  i  and  r,  then  the  expert  prefers  not  to  state  probabilities  r 
with  r(  ^  Pi  when  he  believes  p. 

•  If  —  a„  <  Rf(p)  <  a,  for  i  —  1,  •  •  • ,  m,  then  Rc  is  strictly  proper  at  p,  and  over  a 
neighborhood  of  p. 

•  Rc  is  strictly  proper  at  all  p  sufficiently  close  to  q. 

Consider  an  example  in  which  the  decision-maker  and  the  expert  are  not  jointly  risk -averse.  A 
simple  case  of  this  arises  when  they  are  both  risk-neutral.  If  they  are  betting  up  to  1000  dollars 
each  on  a  coin  flip  and  the  decision-maker  considers  the  coin  fair,  then 

j(-1000,  1000)  if  p  <  5 
/?(/»)  —  {<1000,  —1000)  if  p  >  5 

is  an  admissible  reward.  So  is 


R(p) 


(-1000,  1000)  if  p  <  .5 
(-500,  500)  if  p  -  .5 
(1000,  -1000)  if  p  >  5 


Note  that  the  admissible  reward  is  not  unique.  Also,  the  expert  may  always  bet  the  1000  dol¬ 
lars  he  has  available,  even  when  he  agrees  with  the  decision-maker.  Finally,  the  admissible 
reward  is  not  strictly  proper,  not  even  when  the  decision-maker  and  the  expert  agree  on  the 
probabilities. 


7.  More  Details 

Definition.  Given  A  £  {1,  •  •  • ,  m)  and  EA  =  (J  Eh  the  events  {£,:/€  A]  are  said  to  be 

I  €  A 

an  uninformative  partition  of  EA  if  the  decision-maker  and  expert  agree  on  the  conditional 
probability  of  £,  given  EA  for  all  /  €  A. 

Consider  A  —  {1,  •••,/«}  so  EA  —  £,  and  {£i,  •  •  • ,  Em\  is  an  uninformative  partition  of  EA 
if  p  -  q,  the  decision-maker  and  the  expert  agree  completely. 

Theorem  3. 

Suppose  that  the  decision-maker  and  the  expert  are  jointly  risk-averse.  Then  the  following 
statements  are  true: 

•  If  {£, :  /  €  A\  is  an  uninformative  partition  of  EA  then  Rfip)  -  Rjip)  for  all  /  and  j  6  A. 

•  If  -au  <  Rfip)  <  a„  for  some  i  and  c,  and  Rfip)  -  Rfip)  for  all  j  €  A,  then 
{£, :  /  €  A)  is  an  uninformative  partition  of  EA. 
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Definition.  Mutually  exclusive,  collectively  exhaustive  events  Du  •  •  ■ ,  D„  are  called  refer* 
ence  events  for  Eu  •••,£*,  if  the  decision-maker  and  the  expert  agree  that  E,  and  Dj  are 
independent  (given  F)  for  /  —  1,  •  •  • ,  «  and  j  —  1,  •••,«. 

The  decision-maker  is  not  just  interested  in  whether  a  major  quake  will  occur,  but  would  also 
like  to  know  its  location.  Let  event  N  be  "the  (first)  major  quake  occurs  in  Northern  Califor¬ 
nia”  and  S  be  "the  (first)  major  quake  occurs  in  Southern  California."  Then  instead  of  partition 
{£,  £}  of  £,  the  decision-maker  wants  partition  {£A\  £S,  £).  The  expert  can  never  suffer  by 
going  to  the  more  detailed  bet,  and  will,  indeed,  prefer  it  when  he  disagrees  with  the  decision¬ 
maker  about  the  conditional  probability  of  the  location  given  that  there  is  an  earthquake. 

In  a  similar  fashion,  the  decision-maker  needs  information  about  the  magnitude  of  a  possible 
quake.  Let  event  Mi  be  "the  (first)  major  quake  has  magnitude  less  than  8,"  and  M%  be  "the 
(first)  major_  quake  has_  magnitude  8  or  higher."  The  expert  is  willing  to  substitute 
{EMi,  EM g,  £)  for  {£,  £},  and  prefers  the  switch  if  he  disagrees  with  the  decision-maker 
about  the  magnitude  of  possible  earthquakes. 

Now  suppose  that  the  decision-maker  needs  both  location  and  magnitude  information  at  the 
same  time.  Two  methods  of  designing  the  multiple  bet  allow  us  to  obtain  either  the  marginal 
distributions  or  the  joint  probability  distribution. 

To  assess  the  marginal  distributions  we  need  two  reference  events.  Let  H  be  "a  flipped  coin 
lands  heads-up”  and  T  be  "a  flipped  coin  lands  tails-up."  Clearly  H  and  T  are  mutually 
exclusive  and  collectively  exhaustive,  and  the  outcome  of  the  flip  is  independent  of  any  earth¬ 
quake.  If  the  events  { EHN ,  EHS,  ETMi ,  ETM^  E\  are  used,  then  the  decision-maker  can 
obtain  the  expert’s  conditional  distributions  fbr  magnitude  and  location  at  the  same  time.  If 
they  both  agree  on  the  probabilities  of  reference  events  H  and  T,  then  the  expert’s  satisfaction 
is  between  that  from  the  two  separate  bets.  He  does  not  reveal  any  of  his  opinions  about  their 
dependence.  As  more  details  about  the  events  are  added  to  the  bet,  the  number  of  evefnts 
grows  slowly. 

To  assess  the  joint  distributions  we  consider  the  events  [ENM%  ENM s,  ESMy,  ESM *  E).  The 
expert  reveals  his  beliefs  about  the  dependence  of  location  and  magnitude.  The  expert  prefers 
this  bet  to  all  of  the  bets  mentioned  so  far,  if  he  disagrees  with  the  decision-maker  about  that 
dependence.  This  is  just  a  second  application  of  Theorem  3.  Note  that  the  event  space  is  the 
product  of  the  magnitude  and  location  events  and  the  number  of  events  can  grow  quickly  as  the 
bet  gets  more  detailed. 

8.  Conclusions  and  Extensions 

A  procedure  has  been  suggested  to  reward  an  expert  in  such  a  way  that  he  is  encouraged  to 
reveal  his  true  beliefs  about  uncertain  quantities  as  probabilities.  There  is  a  negotiated  fixed 
payment  to  him  in  exchange  for  his  research  time  and  effort  and,  in  addition,  he  bets  against 
the  decision-maker.  The  expert  prefers  such  a  reward  over  the  fixed  payment  alone,  whenever 
he  disagrees  with  the  decision-maker’s  probabilities.  The  more  they  disagree,  the  more  the 
expert  expects  to  gain  from  the  reward.  The  decision-maker,  on  the  other  hand,  also  prefers  to 
uncover  those  disagreements,  since  he  is  interested  in  learning  from  the  expert. 

Once  the  decision-maker  obtains  the  expert's  probabilities,  he  can  use  Bayes'  Law  to  revise  his 
own  opinion  of  the  uncertain  quantities.  Let  P(  £,  I  F )  be  the  decision-maker's  prior  proba¬ 
bility  for  the  event  E,  given  F.  (Note  that  his  stated  probabilities  q  are  not  necessarily  his  true 
beliefs.)  Let  Pi  p  !  Eh  q  )  be  his  likelihood  function  for  the  expert’s  stated  probabilities  p. 
The  decision-maker’s  q  is  included  because  the  expert  may  learn  from  it.  Bayes’  Law  for  the 
decision-maker’s  posterior  probability  is  then 

P(  E,\  F,  p,  q  )  «  P(  p  |  £„  q  )  Pi  £,  \  F)  . 

The  decision-maker  may  wish  to  consult  with  more  than  one  expert.  This  can  be  addressed  by 
applying  the  procedure  sequentially.  In  this  way,  his  posterior  distribution  after  each  expert  can 
be  used  as  his  prior  distribution  for  the  next.  It  can  be  used  to  determine  whom,  if  anyone,  he 


should  consult  next  and  in  the  design  of  the  next  expert's  reward.  The  likelihood  for  each 
expert  changes  as  other  experts  are  consulted.  First,  one  would  expect  that  experts  on  the 
same  issues  would  be  correlated,  so  that  the  decision-maker  learns  about  the  next  expert  from 
the  previous  ones  [15],  Likewise,  since  the  decision-maker  incorporates  the  opinions  of  the 
experts  already  consulted,  the  next  expert  may  have  more  confidence  in  the  decision-maker’s 
updated  distribution. 

The  procedure  is  designed  to  assess  probability  for  a  finite  number  of  disjoint  events.  That  is 
extended  to  the  case  of  overlapping  events  to  obtain  multiple  marginal  distributions.  Another 
extension  would  be  to  an  uncountable  sample  space.  If  both  the  decision-maker  and  expert 
express  probability  density  functions  which  are  nonzero,  bounded  and  continuous  over  the  sam¬ 
ple  space,  then  the  admissible  reward  is  of  the  same  form  as  for  finite  events. 
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